Assessing the Generalizability of a Performance Predictive Model

A key component of automated algorithm selection and configuration, which in most cases are performed using supervised machine learning (ML) methods is a good-performing predictive model. The predictive model uses the feature representation of a set of problem instances as input data and predicts the algorithm performance achieved on them. Common machine learning models struggle to make predictions for instances with feature representations not covered by the training data, resulting in poor generalization to unseen problems. In this study, we propose a workflow to estimate the generalizability of a predictive model for algorithm performance, trained on one benchmark suite to another. The workflow has been tested by training predictive models across benchmark suites and the results show that generalizability patterns in the landscape feature space are reflected in the performance space.Comment: To appear at GECCO 202

Sensitivity analysis of multilayer neural networks

Dissertation (Ph.D.) -- University of Stellenbosch, 1999.ENGLISH ABSTRACT: The application of artificial neural networks to solve classification and function approximation problems is no longer an art. Using a neural network does not simply imply the presentation of a data set to the network and relying on the so-called "black-box" to produce - hopefully accurate - results. Rigorous mathematical analysis now provides a much better understanding of what is going. on inside the "black-box". The knowledge gained from these mathematical studies allows the development of specialized tools to increase performance, robustness and efficiency. This thesis proposes that sensitivity analysis of the neural network output function be used to learn more about the inner working of multilayer feedforward neural networks. New sensitivity analysis techniques are developed to probe the knowledge embedded in the weights of networks, and to use this knowledge within specialized sensitivity analysis algorithms to improve generalization performance, to reduce learning and model complexity, and to improve convergence performance. A general mathematical model is developed which uses first order derivatives of the neural network output function with respect to the network parameters to quantify the effect small perturbations to these network parameters have on the output of the network. This sensitivity analysis model is then used to develop techniques to locate and visualize decision boundaries, and to determine which boundaries are implemented by which hidden units. The decision boundary detection algorithm is then used to develop an active learning algorithm for classification problems which trains only on patterns close to decision boundaries. Patterns that convey little information about the position of boundaries are therefore not used for training. An incremental learning algorithm for function approximation problems is also developed to incrementally grow the training set from a candidate set by adding to the training set those patterns that convey the most information about the function to be approximated. The sensitivity of the network output to small perturbations of the input pattern is used as measure of pattern informativeness. Sensitivity analysis is also used to develop a network pruning algo-rithm to remove irrelevant network parameters. The significance of a parameter is quantified as the influence small perturbations on that parameter have on the network output. Variance analysis is employed as pruning heuristic to decide if a parameter should be removed or not. Elaborate experimental evidence is provided to illustrate how each one of the developed sensitivity analysis techniques addresses the objectives of improved performance, robustness and efficiency. These results show that the different models successfully utilize the neural network learner's current knowledge to obtain optimal architectures and to make optimal use of the available training data.AFRIKAANSE OPSOMMING: Die toepassing van kunsmatige neurale netwerke om klassifikasie- en funksiebenaderingsprobleme op te los, is nie meer 'n kuns nie. Die gebruik van 'n neurale netwerk impliseer nie meer bloot die toepassing van 'n data stel op die netwerk, en die verwagting dat die "swart boks" - hoopvol akkurate - result ate lewer nie. Omvattende wiskundige analises verskaf nou 'n baie beter begrip van wat binne die "swart boks" aangaan. Die kennis wat van hierdie wiskundige analises gewin is, laat die ontwikkeling van gespesialiseerde hulpmiddels toe om prestasie, robuustheid en effektiwiteit te verbeter. Hierdie tesis stel voor dat sensitiwiteitsanalise van die neurale netwerk afvoer funksie aangewend word om meer oor die inner werking van multi-vlak vorentoe-voer neurale netwerke te leer. Nuwe sensitiwiteitsanalise tegnieke word ontwikkel om die kennis vervat in die gewigte van netwerke te ondersoek, en om hierdie kennis aan te wend binne gespesialiseerde sensitiwiteitsanalise algoritmes om sodoende veralgemeningseienskappe te verbeter, om die kompleksiteit van leer en model kompleksiteit te verminder, en om konvergensie eienskappe te verbeter. 'n Algemene wiskundige model is ontwikkel wat gebruik maak van die eerste orde afgeleides van die neurale netwerk afvoer funksie met betrekkiIig tot netwerk parameters om die effek van klein versteurings aan hierdie netwerk parameters op die afvoer van die rietwerk te kwantifiseer. Hierdie sensitiwiteitsanalise model word dan gebruik om tegnieke te ontwikkel om besluitnemingsgrense op te spoor en te visualiseer, en om te bepaal watter besluitnemingsgrense word deur watter versteekte eenhede geimplementeer. Die algoritme om besluitnemingsgrense op te spoor word dan aangewend om 'n aktiewe-leer algoritme vir klassifikasie probleme te ontwikkel, wat leer deur gebruik te maak van slegs daardie patrone wat naby besluitnemingsgrense Ie. Gevolglik word patrone wat min inligting bevat in verband met die ligging van besluitnemingsgrense nie vir leer aangewend nie. 'n Inkrementele leer algoritme is ook ontwikkel vir funksiebenaderingsprobleme waarin die leerversameling inkrementeel vanuit 'n kandidaat leerversameling gegroei word deur daardie patrone by te voeg wat die meeste inligting vervat oor die funksie wat benader word. Die sensitiwiteit van die netwerk afvoer tot versteurings in die toevoer patroon word gebruik as 'n maatstaf van die informatiwiteit van daardie patroon. Sensitiwiteitsanalise is ook gebruik om 'n algoritme te ontwikkel wat irrelevante parameters van die netwerk snoei. Die belangrikheid van 'n parameter word gekwantifiseer as die invloed wat klein versteurings in daardie parameter het op die afvoer van die netwerk. Variansie analise word gebruik as heuristiek om te besluit of 'n parameter gesnoei kan word al dan nie. Omvattende eksperimentele bewyse word verskaf om te illustreer hoe elkeen van die sensitiwiteitsanalise tegnieke wat in hierdie tesis ontwikkel is, die doelwitte van verbeterde prestasie, robuustheid en effektiwiteit adresseer. Hierdie resultate toon aan dat die onderskeie modelle suksesvol gebruik maak van die neurale netwerk se huidige kennis om optimale argitekture op te stel, en om optimaal van die beskikbare leerdata gebruik te maak

An Exploratory Landscape Analysis-Based Benchmark Suite

The choice of which objective functions, or benchmark problems, should be used to test an optimization algorithm is a crucial part of the algorithm selection framework. Benchmark suites that are often used in the literature have been shown to exhibit poor coverage of the problem space. Exploratory landscape analysis can be used to quantify characteristics of objective functions. However, exploratory landscape analysis measures are based on samples of the objective function, and there is a lack of work on the appropriate choice of sample size needed to produce reliable measures. This study presents an approach to determine the minimum sample size needed to obtain robust exploratory landscape analysis measures. Based on reliable exploratory landscape analysis measures, a self-organizing feature map is used to cluster a comprehensive set of benchmark functions. From this, a benchmark suite that has better coverage of the single-objective, boundary-constrained problem space is proposed

An Exploratory Landscape Analysis-Based Benchmark Suite

The choice of which objective functions, or benchmark problems, should be used to test an optimization algorithm is a crucial part of the algorithm selection framework. Benchmark suites that are often used in the literature have been shown to exhibit poor coverage of the problem space. Exploratory landscape analysis can be used to quantify characteristics of objective functions. However, exploratory landscape analysis measures are based on samples of the objective function, and there is a lack of work on the appropriate choice of sample size needed to produce reliable measures. This study presents an approach to determine the minimum sample size needed to obtain robust exploratory landscape analysis measures. Based on reliable exploratory landscape analysis measures, a self-organizing feature map is used to cluster a comprehensive set of benchmark functions. From this, a benchmark suite that has better coverage of the single-objective, boundary-constrained problem space is proposed

Function minimization in DNA sequence design based on continuous particle swarm optimization

In DNA based computation and DNA nanotechnology, the design of good DNA sequences has turned out to be an essential problem and one of the most practical and important research topics. Basically, the DNA sequence design problem is a multi-objective problem, and it can be evaluated using four objective functions, namely, Hmeasurei similarity, continuity, and hairpin. In this paper, particle swarm optimization (PSO) is proposed to minimize those objective functions, individually, subjected to two constraints: melting temperature, Tm, and GCcontent. A model is presented in order to minimize the objective functions using PSO. An implementation of the optimization process is presented using 20 particles. The results obtained verified that PSO can be used to minimize each objective individually